Exploring the Contextual Factors Affecting Multimodal Emotion Recognition in Videos

Emotional expressions form a key part of user behavior on today's digital platforms. While multimodal emotion recognition techniques are gaining research attention, there is lack deeper understanding how visual and non-visual features can be used to better recognize emotions in certain contexts, but not others. This study analyzes the interplay between effects derived from facial expressions, tone text conjunction with two contextual factors: 1) gender speaker, 2) duration emotional episode. Using large public dataset 2,176 manually annotated YouTube videos, we found that while consistently outperformed bimodal unimodal features, their performance varied significantly across different emotions, contexts. Multimodal performed particularly for male speakers recognizing most emotions. Furthermore, shorter than longer videos neutral happiness, sadness anger. These findings offer new insights towards development more context-aware empathetic systems.